Summarization of short comments

ABSTRACT

A method and a system for summarization of short comments are provided. The system comprises a memory to store a plurality of comments. Each of the plurality of comments including an overall rating and at least one phrase. The system also includes one or more processors to implement an aspect module to identify a head term based on a portion of the plurality of comments and to map the portion of the plurality of comments to an aspect corresponding to an attribute of an entity. The one or more processor also implement an extraction module to extract, from the portion of the plurality of comments, a representative phrase corresponding to the aspect.

This application is a continuation of U.S. patent application Ser. No.14/242,505, filed on Apr. 1, 2014 and entitled “Summarization of ShortComments”, which is a continuation of U.S. patent application Ser. No.12/766,697, filed on Apr. 23, 2010 and entitled “Summarization of ShortComments” now U.S. Pat. No. 8,713,017 issued on Apr. 29, 2014, whichclaims the priority benefit of U.S. Provisional Application No.61/172,151, filed Apr. 23, 2009 and entitled “Rated Aspect Summarizationof Short Comments,” which are incorporated herein by reference.

TECHNICAL FIELD

The present application relates generally to the technical field ofinformation management and, in one specific example, to summarization ofshort comments.

BACKGROUND

Websites allow users to contribute and receive information. For example,people comment on various entities such as sellers, products andservices. These user-contributed comments, in turn, educate other usersand drive traffic to the website. Generally, given a target entity, manyuser-generated short comments are received. Each of these comments mayinclude an overall rating. For example, users may review and rateproducts on websites such as CNET using one to five stars. On anotherwebsite, such as eBay, buyers leave feedback comments to the seller andrate the transaction as positive, neutral, or negative. Usually, thenumber of comments about a target entity numbers in the hundreds orthousands. This number is constantly growing as more and more peoplecontribute more comments.

BRIEF DESCRIPTION OF THE DRAWINGS

Some embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings in which:

FIG. 1 is a depiction of an example input and example output of asummarization engine, according to an example embodiment.

FIG. 2 is a block diagram of a summarization engine, according tovarious embodiments.

FIG. 3 is a block diagram of an aspect module, according to variousembodiments.

FIG. 4 is a block diagram of a rating module, according to variousembodiments.

FIG. 5 is a block diagram of an evaluation module, according to variousembodiments.

FIG. 6 is a graph depicting sample results of an evaluation of aspectcoverage.

FIG. 7 is a graph depicting a sample human agreement curve of clusteringaccuracy.

FIG. 8 is a flowchart of a technique for summarizing the short comments,according to various embodiments.

FIG. 9 is a network diagram depicting a client-server system withinwhich one example embodiment may be deployed.

FIG. 10 shows a diagrammatic representation of machine in the exampleform of a computer system within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed.

DETAILED DESCRIPTION

Example methods and systems to summarize short comments are described.In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of example embodiments. It will be evident, however, toone skilled in the art that the present invention may be practicedwithout these specific details.

FIG. 1 is a depiction of an example input and example output of asummarization engine according to an example embodiment. The exampleinput 100 includes a number of comments 102 (630,748 comments as shown)and the example output 106 includes a rated aspect summary wherecomments are decomposed into several aspects 108 and each aspect hasassociated support information (shown as stars 110). More specifically,the input 100 represents the data that a user typically views in acommunity comments website. This data consists of a large number ofcomments 102, each having companion overall ratings 104. Using suchdata, the user can only get an overall impression by looking at theaverage overall rating. The summarization engine described hereingenerates the output 106, which comprises a rated aspect summary that,in turn, provides the decomposed overall ratings (e.g., stars 110) ofthe rated aspects 108, so that a user can gain different perspectivestowards a target entity. This kind of the decomposition is quite usefulbecause differing users may have different needs and the overall ratingsare generally not tailored to satisfy those needs. For example, aprospective eBay buyer may compromise on shipping time but not onproduct quality. In this case, it is not sufficient for the buyer tojust know the overall rating of the seller, and it would be highlydesirable for the buyers to know the ratings are based on a specificaspect about product quality.

The output 106 shows the overall rating is being decomposed into severalrated aspects 108, such as shipping communication and service. Eachaspect has support information shown by the stars 110. Additionally,representative phrases 112 with support information (e.g., bars 114) areincluded with the rated aspects 108, and can serve as indices tonavigate into a set of specific comments about a particular aspect.

The rated aspect summarization may be helpful even when users explicitlyprovide ratings for a given aspect. For example, it may be desirable tofurther decompose the aspect into sub-aspects. To illustrate, peopletypically rate “food” in restaurant reviews, but users usually want toknow in what sense the food was “good” or “bad.” For example. aparticular user may be concerned more about healthiness than abouttaste. In other instances, the given aspects may not cover all of themajor aspects discussed in the text comments. For example, in the eBaysystem, there are four defined aspects to rate a seller, namely, “itemas described,” “communication,” “shipping time,” and “shipping andhandling charges.” It would be difficult to know the seller'sperformance on aspects such as “packaging,” “price,” or “service,” whichmay be useful to some potential buyers.

To provide rated aspect summaries, a summary engine first extracts majoraspects from comments in a collection and then predicts a rating foreach aspect from the overall ratings. Finally, the summarization engineextracts representative phrases from the short comments.

In particular, the embodiments described here generate a rated aspectsummary (i.e., a summary with a rating for each aspect) in order to helpusers better comprehend the comments along different dimensions of thetarget entity given a large number of short comments about the targetentity, where each comment is associated with an overall ratingindicating an overall opinion about the target entity. There are twodifferent scenarios. The first scenario is without supervision wherethere is no prior knowledge of the aspects. The aspects areautomatically decomposed from the overall rating into purely ad hocaspects based on the data. In a second scenario, using minimumsupervision, the user provides keywords specifying aspects of interestand those aspects are accommodated.

Some definitions referred to herein include:

The collection of short comments is denoted by T={t₁, t₂, . . . }, whereeach short comment t is an element of the comments collection T (tϵT)and is associated with an overall rating r(t).

An overall rating r(t) of a comment t is a numerical rating indicatingdifferent levels of overall opinion contained in comment t, the ratingis a value between a predefined minimum value and a predefined maximumvalue (r(t)ϵ{r_(min), . . . , r_(max)}. Usually, it is infeasible for auser to review the overall ratings of a large number of comments.

The average overall rating of a collection of comments R(T) is a scoreaveraged over all of the overall ratings expressed as:

${R(T)} = {\frac{\Sigma_{t \in T}{r(t)}}{T} \in \left\lbrack {r_{\min},r_{\max}} \right\rbrack}$

A phrase f=(w_(m),w_(h)) is in the form of a pair of head term (w_(h))and modifier (w_(m)). Usually the head term is an aspect or feature, andthe modifier expresses some opinion about this aspect. In shortcomments, such as feedback text, most opinions are expressed in concisephrases, such as “well packaged,” “excellent seller.” Using shallowparsing techniques, those phrases are extracted to identify the headterm and the modifier.

Each comment is represented by a bag of phrases(t={f=(w_(m),w_(h))|fϵt}) instead of a bag of words. The rated aspectsummarization includes three steps: identify a number (k) of major“aspect clusters,” predict an “aspect rating” for each aspect, andextract “representative phrases” to support or explain the aspectratings. These concepts are defined herein as follows:

An aspect cluster (A_(i)) is a cluster of head terms (w_(h)) that sharesimilar meaning in the given context. Those words jointly represent anaspect that users comment on and/or are of interest to users. The aspectcluster is denoted as A_(i)={w_(h)|A(w_(h))=i}, where A(.) is a mappingfunction from some aspect clustering algorithm that maps a head term toa cluster label.

An aspect rating (R(A_(i))) is a numerical measure with respect to theaspect A, showing the degree of satisfaction demonstrated in thecomments collection T toward this aspect, and R(A_(i))ϵ[r_(min), . . . ,r_(max)].

A representative phrase rf=(f, s(f)) is a phrase f with a support values(f), where s(f)ϵ[1,∞) indicating how many phrases in the comments thatthis phrase can represent.

Note that r(.) is used to denote a discrete rating (an integer betweenr_(min) and r_(max)), and R(.) is used to denote an average rating overa number of discrete ratings, which is a rational number (usuallynon-integer) between r_(min) and r_(max).

A rated aspect summary is a set of tuples expressed as:(A _(i) ,R(A _(i)),RF(A _(i)))_(i) ^(k)=1where A_(i) is a ratable aspect, R(A_(i)) is the predicted rating onA_(i), and RF(A_(i)) is a set of representative phrases in this aspect.

FIG. 2 is a block diagram of a summarization engine 200 according tovarious embodiments. The summarization engine 200 comprises (or hasaccess to) a comments collection 202, an aspect module 204, a ratingmodule 206, an extraction module 208, and an optional evaluation module210. The summarization engine 200 summarizes short comments about atarget entity.

The short comments are stored in a comments collection 202 and can beaccessed by the summarization engine 200. The short comments may includeseller feedback, customer reviews, and comments on content (e.g., blogs,new articles, etc.) about a target entity.

The aspect module 204 is used to identify a number k of aspects and tocluster head terms into those aspects. The short comments includeopinions on different aspects that, in turn, are usually expressed asconcise phrases. Each phrase f is parsed into a pair of head terms,w_(h), and modifier terms, w_(m), in the form of f=(w_(m),w_(h)).Usually the head term, w_(h), is about an aspect or feature of thetarget entity, and the modifier expresses some opinion towards thisaspect. The aspect module 204 may use a number of techniques to identifyand cluster aspects. Three examples of these techniques are discussed ingreater detail in connection with FIG. 3.

The rating module 206, once the k aspect clusters of head terms areidentified by the aspect module 204 in the form of a clustering mappingfunction A(.), predicts the rating for each aspect from the overallrating. The rating module 206 may use a variety of techniques forclassifying each phrase f into a rating r(f) having the same scale asthe overall ratings. The aspect ratings are, in turn, calculated byaggregating ratings of the phrases within each aspect. Exampletechniques that may be used are described more fully in connection withFIG. 4.

The extraction module 208 extracts representative phrases from thecomments in order to provide the users with some textual clues forbetter understanding of the predicted aspect rating. If the aspectclusters and aspect rating predictions are accurate, the phrases thatare classified into the same aspect and same rating are similar to oneanother. In these cases, the comments collection T is segmented intosubsets of phrases F for each aspect A_(i) and each rating value r, andF(A _(i) ,r)={f|A(f)=i,r(f)=r}expressed as:Using the segmented phrases, the top three phrases with the highestfrequency in each subset are extracted. The support value for a phrase fis the frequency of the phrase in the subset, expressed as:where c is a function to determine the number of co-occurrences of thephrase f ins(f)=c(f,F(A _(i) ,r))the subset of phrases F.

The optional evaluation module 210 is to evaluate the performance of theaspect module 204, the rating module 206, and the extraction module 208.In some instances, the resulting aspects, ratings, and extracted phrasesare compared to an existing set of ratings, aspects and extractedphrases. The evaluation module 210 is discussed in greater detail inconnection with FIG. 5.

FIG. 3 is a block diagram of an aspect module 204, according to variousembodiments. The aspect module 204 comprises a k-means calculator 302,an unstructured probabilistic latent semantic analysis (PLSA) calculator304, a structured PLSA calculator 306, and an aspect estimator 308.

The k-means calculator 302 is to apply a k-means standard clusteringalgorithm to a set of vectors of head terms. The k-means standardclustering algorithm may be used, in part, because the structure ofphrases may be used in the clustering of the terms. To illustrate, iftwo head terms tend to be associated with the same set of modifiers, thehead terms may share similar meaning. For example, head terms that areusually modified by “fast” should be more similar to one anotherv(w _(h))=(c(w _(h) ,w _(m) ¹),c(w _(h) ,w _(m) ²), . . . )than head terms modified by “polite” or “honest.” In some instances, therelation between modifiers is used by representing each of the headterms w_(h) as a vector v(w_(h)) in the form ofwhere c(w_(h),w_(m) ^(i)) is the number of co-occurrences of head termw_(h) with modifier w_(m) ^(i). The k-means clustering algorithm is thenapplied to the vector. The clusters output by k-means are used as theaspects of interest. K-means clustering is a statistical and machinelearning technique for clustering or partitioning a group of nobservations or objects into k clusters in which each observation orobject belongs to one of the k clusters with the nearest mean or acomputed average of the distance between observations or objects in thecluster.

The unstructured PLSA calculator 304 provides an alternate or additionalway to identify aspects or clusters of aspects from the short comments.By applying the PLSA algorithm to head terms in unstructured texts,aspects can be extracted. Generally, a unigram language model (i.e., amultinomial word distribution) is used to model a topic. For example, adistribution that assigns high probabilities to words such as“shipping,” “delivery,” and “days,” may indicate a topic such as“shipping time.” In order to identify multiple aspects in text, amixture model involving multiple multinomial distributions may be fit tothe text data. The mixture model may also be used to determine theparameters of the multiple word distribution so that the likelihood ofthe text data is maximized.

A number k of unigram language models are defined as Θ={θ₁,θ₂, . . . ,θ_(k)} as k theme models, each being a multinomial distribution of headterms, capturing one aspect. A comment tϵT may be regarded as a sampleof the

${p_{t}\left( w_{h} \right)} = {\sum\limits_{j = 1}^{k}\;\left\lbrack {\pi_{t,j}{p\left( w_{h} \middle| \theta_{j} \right)}} \right\rbrack}$mixture model:where w_(h) is a head term, π_(t,j) is a comment-specific mixture weightfor the j-th aspect, which is represented as:Σ_(j=1) ^(k)π_(t,j)=1The log-likelihood of the collection T is calculated by:

${\log\;{p\left( T \middle| \Lambda \right)}} = {\sum\limits_{t \in T}\mspace{14mu}{\sum\limits_{w_{h} \in V_{h}}\mspace{14mu}\left\{ {{c\left( {w_{h},t} \right)} \times \log{\sum\limits_{j = 1}^{k}\;\left\lbrack {\pi_{t,j}{p\left( w_{h} \middle| \theta_{j} \right)}} \right\rbrack}} \right\}}}$where V_(h) is the set of all head terms, c(w_(h),t) is the count of thehead term w_(h) comment t, and Λ is the set of all model parameters.

The model can be estimated using any estimator. For example, theExpectation-Maximization (EM) algorithm can be used to compute a maximumlikelihood estimate with the following updating formulas:

${p\left( z_{t,w_{h},j} \right)} = \frac{\pi_{t,j}^{(n)}{p^{(n)}\left( w_{h} \middle| \theta_{j} \right)}}{\sum_{j^{\prime} = 1}^{k}{\pi_{t,j^{\prime}}^{(n)}{p^{(n)}\left( w_{h} \middle| \theta_{j^{\prime}} \right)}}}$$\pi_{t,j}^{({n + 1})} = \frac{{\Sigma\;}_{w_{h} \in V_{h}}\mspace{14mu}{c\left( {w_{h},t} \right)}{p\left( z_{t,w_{h},j} \right)}}{\Sigma_{j^{\prime}}\Sigma_{w_{h} \in V_{h}}\mspace{14mu}{c\left( {w_{h},t} \right)}{p\left( z_{t,w_{h},j^{\prime}} \right)}}$${p^{({n + 1})}\left( w_{h} \middle| \theta_{j} \right)} = \frac{\Sigma_{t \in T}\mspace{14mu}{c\left( {w_{h},t} \right)}{p\left( z_{t,w_{h},j} \right)}}{\Sigma_{w_{h}^{\prime} \in V_{h}}\mspace{14mu}\Sigma_{t \in T}\mspace{14mu}{c\left( {w_{h}^{\prime},t} \right)}{p\left( z_{t,w_{h}^{\prime},j} \right)}}$where p(z_(t,wh,j)) represents the probability of head term w_(h) incomment t assigned to the jth aspect.

A set of theme models may be extracted from the text collection{θ_(i)|i=1, . . . , k}, and each head term w_(h)ϵV_(h) is grouped intoone of k aspects by selecting the theme model with the largestprobability of generating w_(h), which is a clustering mapping function:

${A\left( w_{h} \right)} = {\underset{j}{\arg\mspace{14mu}\max}\mspace{14mu}{p\left( w_{h} \middle| \theta_{j} \right)}}$If two head terms tend to co-occur with each other (such as “ship” and“delivery” co-occurring in “fast ship and delivery”) and one term isassigned a high probability, then the other generally is also assigned ahigh probability in order to maximize the data likelihood. Thus, theunstructured PLSA calculator 304 generally captures the co-occurrencesof head terms and may cluster the head terms into aspects based onco-occurrences in comments.

The structured PLSA calculator 306 incorporates the structure of phrasesinto the PLSA model using co-occurrence information of head terms andtheir modifiers. A number k of unigram language models is defined asΘ={θ₁,θ₂, . . . , θ_(k)} as k theme models. Each modifier term isrepresented by a set of head terms that it modifies:d(w _(m))={w _(h)|(w _(m) ,w _(h))ϵT}which is regarded as a sample of the mixture model:

${p_{d{(w_{m})}}\left( w_{h} \right)} = {\sum\limits_{j = 1}^{k}\;\left\lbrack {\pi_{{d{(w_{m})}},j}{p\left( w_{h} \middle| \theta_{j} \right)}} \right\rbrack}$where π_(d(wm),j) is a comment-specific mixture weight for the j-thaspect, which sumsΣ_(j=1) ^(k)π_(d(w) _(m) _(),j)=1to one and is represented as:log p(V _(m)|Λ)=Σ_(w) _(m) _(ϵV) _(m) _(Σw) _(h) _(ϵV) _(h) {c(w _(h),d(w _(n)))×log Σ_(j=1) ^(k)[π_(d(w) _(m) _(),j) p(w _(h)|θ_(j))]}The log-likelihood of the collection of modifiers V_(m) is:where c(w_(h), d(w_(m))) is the number of co-occurrences of head termw_(h) with modifiers w_(m), and A is the set of all model parameters.Using a similar EM algorithm as the unstructured PLSA calculator 304,the k theme models are estimated and the clustering mapping function isobtained. For completeness, the updating formulas

${p\left( z_{{d{(w_{m})}},w_{h},j} \right)} = \frac{\pi_{{d{(w_{m})}},j}^{(n)}{p^{(n)}\left( w_{h} \middle| \theta_{j} \right)}}{\sum_{j^{\prime} = 1}^{k}{\pi_{{d{(w_{m})}},j^{\prime}}^{(n)}{p^{(n)}\left( w_{h} \middle| \theta_{j^{\prime}} \right)}}}$$\pi_{{d{(w_{m})}},j}^{({n + 1})} = \frac{{\Sigma\;}_{w_{h} \in V_{h}}\mspace{14mu}{c\left( {w_{h},{d\left( w_{m} \right)}} \right)}{p\left( z_{{d{(w_{m})}},w_{h},j} \right)}}{\Sigma_{j^{\prime}}\Sigma_{w_{h} \in V_{h}}\mspace{14mu}{c\left( {w_{h},{d\left( w_{m} \right)}} \right)}{p\left( z_{{d{(w_{m})}},w_{h},j^{\prime}} \right)}}$${p^{({n + 1})}\left( w_{h} \middle| \theta_{j} \right)} = \frac{\Sigma_{w_{m} \in V_{m}}\mspace{14mu}{c\left( {w_{h},{d\left( w_{m} \right)}} \right)}{p\left( z_{{d{(w_{m})}},w_{h},j} \right)}}{\Sigma_{w_{h}^{\prime} \in V_{h}}\mspace{14mu}\Sigma_{w_{m} \in V_{m}}\mspace{14mu}{c\left( {w_{h}^{\prime},{d\left( w_{m} \right)}} \right)}{p\left( z_{{d{(w_{m})}},w_{h}^{\prime},j} \right)}}$follow:where p(z_(d(wm),wh,j)) represents the probability of head term w_(h)associated with the modifier w_(m) assigned to the jth aspect.

Relative to the unstructured PLSA calculator 304, the structured PLSAcalculator 306 models the co-occurrence of head terms at the level ofthe modifiers used instead of at the level of comments that occur. Incontrast, the structured PLSA calculator 306 organizes the head terms bytheir modifiers.

The aspect estimator 308 incorporates domain knowledge about theaspects. For instance, “food” and “service” are major aspects incomments about restaurants. And sometimes a user may have specificpreference on some aspects. For example, a buyer may be especiallyinterested in the “packaging” aspect. In the probabilistic modelframework, a conjugate prior may be used to incorporate such humanknowledge to guide the clustering of aspects.

Specifically, a unigram language model {p(w_(h)|a_(j))}_(Whϵvh) is builtfor each aspect that is associated with some prior knowledge. Forexample, a language model for a “packaging” aspect may look like:p(packaging|a ₁)=0.5p(wrapping|a ₁)=0.5A conjugate prior (i.e., a Dirichlet prior) is defined on each unigramlanguage model, parameterized as:Dir({σ_(i) p(w _(h) |a _(j))+1}_(w) _(h) _(ϵV) _(h)where σ_(j) is a confidence parameter for the conjugate prior. Since aconjugate prior is used, σ_(j) is interpreted as the “equivalent samplesize,” which means that the effect of adding the conjugate prior wouldbe equivalent to adding σ_(j) p(w_(h)|a_(j))+1 pseudo counts for headterm w_(h) when the topic model p(w_(h)|θ_(j)) is estimated. Basically,the conjugate prior serves to bias the clustering results.

The prior for all the parameters is given by:

${p(\Lambda)} \propto {\prod\limits_{j = 1}^{k}\;{\underset{w_{h} \in V_{h}}{\Pi}\mspace{14mu}{p\left( w_{h} \middle| \theta_{j} \right)}^{\sigma_{j}{p{({w_{h}|a_{j}})}}}}}$where σ_(j)=0 if there is no prior knowledge about some aspect θ_(j).

The Maximum A Posterioi (MAP) estimator is used to estimate all theparameters as follows (for unstructured PLSA and structured PLSA,

$\hat{\Lambda} = {\underset{\Lambda}{\arg\mspace{14mu}\max}\mspace{14mu}{p\left( T \middle| \Lambda \right)}{p(\Lambda)}}$$\hat{\Lambda} = {\underset{\Lambda}{\arg\mspace{14mu}\max}\mspace{14mu}{p\left( V_{m} \middle| \Lambda \right)}{p(\Lambda)}}$respectively):The MAP estimate is computed using essentially the same EM algorithmusing modified updating formulas for the component language models asfollows (for unstructured and structured PLSA calculations,respectively):and

${p\left( w_{h} \middle| \theta_{j} \right)}^{({n + 1})} = \frac{{\Sigma_{w_{m} \in V_{m}}\mspace{14mu}{c\left( {w_{h},{d\left( w_{m} \right)}} \right)}{p\left( z_{{d{(w_{m})}},w_{h},j} \right)}} + {\sigma_{j}{p\left( w_{h} \middle| a_{j} \right)}}}{{\Sigma_{w_{h}^{\prime} \in V_{h}}\Sigma_{w_{m} \in V_{m}}\mspace{14mu}{c\left( {w_{h}^{\prime},{d\left( w_{m} \right)}} \right)}{p\left( z_{{d{(w_{m})}},w_{h}^{\prime},j} \right)}} + \sigma_{j}}$

FIG. 4 is a block diagram of a rating module 206, according to variousembodiments. The rating module 206 predicts the rating for the aspectsidentified by the aspect module 204 of FIG. 3 from the overall ratingassociated with each of the short comments. The rating module 206 usestwo or more techniques to classify each phrase f into a rating r(f)based the same scale as the overall ratings. The aspect ratings arecalculated by aggregating ratings of the phrases within each aspect. Therating module 206 comprises a local predictor 402, a global predictor404 and an aggregator 406.

The local predictor 402 uses the local information (i.e., the overallrating of the exact comment in which a particular phrase appears)associated with the comment. The technique used by the local predictor402 is based on an assumption that an overall rating given by a user isconsistent with what is written in the comment. In this technique, eachphrase in a given short comment is assigned the same rating as theoverall rating of the comment. Expressed mathematically, the ratingclassifier for a phrase is:r(fϵt)=r(t)ϵ{r _(min) , . . . , r _(max)}

The global predictor 404 learns rating level classifiers using globalinformation of the overall ratings of all comments. The global predictor404 classifies each phrase by the globally learned rating classifier.Specifically, for each aspect A_(i), r_(max)−r_(min)+1 rating models areestimated empirically, each corresponding to a rating value rϵ{r_(min),. . . , r_(max)}. Each rating model is a unigram

${p\left( {\left. w_{m} \middle| A_{i} \right.,r} \right)} = \frac{c\left( {w_{m},{S\left( {A_{i},r} \right)}} \right)}{\Sigma_{w_{m}^{\prime} \in V_{m}}\mspace{14mu}{c\left( {w_{m}^{\prime},{S\left( {A_{i},r} \right)}} \right)}}$language model of modifiers capturing the distribution of modifiers withthe given rating value. The rating value model is estimated by theempirical distribution:whereis a subset of phrases that belong to this aspect, and commentscontaining these phrases receive the overall rating of r. Each phrasecan then be classified byS(A _(i) ,r)={f|fϵt,A(f)=i, and r(t)=r}choosing the rating class that has the highest probability of generatingthe modifier in that phrase. This may be considered to be a Naïve Bayesclassifier with uniform

${r(f)} = {\underset{r}{\arg\mspace{14mu}\max}\mspace{14mu}\left\{ {\left. {p\left( {\left. w_{m} \middle| A_{i} \right.,r} \right)} \middle| {A(f)} \right. = i} \right\}}$prior on each rating class, expressed as:Intuitively, the rating class supplier of global prediction should workbetter than that of local prediction. In some cases, not all the phrasesin a comment are consistent with the overall rating. It is quitepossible that a user will give a high overall rating while mentioningsome shortcomings in a comment and vice-versa. Suppose the comments have“slow shipping” rated as a maximum score, local prediction would blindlyrate to phrase a maximum score but a global prediction could potentiallyidentify that “slow” is a low rating for the aspect “shipping,” because“slow” typically appears in more comments about shipping having a lowerrating than in comments having a higher rating.

Once each phrase is classified into a rating value by the localpredictor 402 and the global predictor 404, the rating for each aspectA_(i) is calculated by the aggregator 406. The aggregator 406 aggregatesthe rating of the phrases that are clustered into the aspect. One waythe ratings can be aggregated is to calculate the average rating ofphrases within this aspect according to the formula:

${R\left( A_{i} \right)} = \frac{\Sigma_{{A{(f)}} = i}{r(f)}}{\left| \left\{ {\left. f \middle| {A(f)} \right. = i} \right\} \right|}$where R(A_(i)) is a value between r_(min) and r_(max), representing theaverage rating of this aspect.

FIG. 5 is a block diagram of an evaluation module 210, according tovarious embodiments. The evaluation module 210 is used to evaluate theaccuracy of the ratings predictions. A dataset is created by collectingfeedback comments for 28 sellers on eBay with high feedback scores overthe past year. The feedback score of a seller is defined as thecumulative number of positive feedback received. After each transaction,the buyer is supposed to leave some feedback for the seller including:an overall rating is positive, neutral or negative; detailed sellerratings (DSRs), on four given aspects: “item as described,”“communication,” “shipping time,” and “shipping and handling charges” ona scale of five stars; and some short comments in free text.

For preprocessing, the part of speech (POS) tagging and chunkingfunction of the open natural language processor (NLP) toolkit is used toidentify phrases in the form of a pair of head term and modifier.Statistics about the dataset is provided in Table 1 below:

TABLE 1 Statistics of the Data Set Statistics Mean STD # of comments perseller 57,055 62,395 # of phrases per comment 1.5533 0.0442 overallrating (positive %) 0.9799 0.0095

Based on these statistics, a few observations can be made. First, thosesellers with high feedback scores receive a large number of comments,57,055 comments on average. But the number also varies across differentsellers, as the standard deviation is very high. Second, buyers usuallyonly use a few phrases in each comment. After parsing there are about1.5 phrases per comment. Note that, the original data is noisier. Forexample, the user-invented superlative “AAA+++” does not provide muchdetailed information on aspects. The preprocessing reduces the data byabout 40% in terms of the number of tokens. Third, the overall averageoverall ratings are usually very high, nearly 0.98 of comments arepositive, so they are not discriminative.

A sample rated aspects summarization of one of the sellers is shown inTable 2, below.

TABLE 2 A Sample Result of Rated Aspect Summarization No. AspectsRatings Phrases of Rating 1 Phrases of Rating 0 1 described, promised4.8457 as described (3993) than expected (68) as promised (323) thandescribed(43) as advertised (149) i ordered (10) 2 shipped, arrived4.3301 quickly shipped (162) open box (39) great thanks (149) wrong sent(29) quickly arrived (138) back sent (15) 3 recommended, was 3.9322highly recommended (236) back be (42) highly recommend (115) defectivewas (40) exactly was (84) not have (37) 4 shipping, delivery 4.7875 fastshipping (5354) good shipping (170) quick shipping (879) slow shipping(81) fast delivery (647) reasonable shipping (32) 5 transaction, item4.6943 great item (1017) wrong item (70) great transaction (704) newcondition (48) smooth transaction (550) new item (34) 6 seller, product4.9392 great seller (2010) poor communication (12) great product (1525)defective product (12) good seller (866) personal comm (9) 7 works,price 4.3830 great works (1158) perfectly works (132) great price (642)fine works (90) good price (283) not working (29) 8 buy, do 4.0917 willbuy (356) not did (105) would buy (347) not work (91) again buy (271)didnt work (49)

The first column of Table 2 shows automatically discovered and clusteredaspects using structured PLSA. The number of aspects is empirically setto eight. The top two head terms in each aspect are displayed as theaspect label. The second column is the predicted ratings for differentaspects using global prediction. Due to the mostly positive nature ofthe feedback, both neutral and negative feedback are assigned a ratingof zero, and positive feedback is assigned a rating of one. Thepredicted ratings for each aspect are a value between zero and one.Then, the predicted ratings are mapped uniformly to the five-starratings to produce a score between zero and five as in the second columnof the table. The last two columns show three representative phrasestogether with their frequency for each aspect and for rating one andzero respectively.

Based on the summarization, a few observations are made. First, that themajor aspects can be discovered and clustered around head terms in ameaningful way. Aspect 1 is about whether the seller truly delivers aspromised; aspect 3 shows whether the buyers would recommend seller;aspect 7 pertains to price. Almost all aspects are coherent andseparable except that aspect 2 and aspect 4 both pertain to “shippingtime.” Second, the aspect ratings provide some insight towards theseller's performance on the different aspects. Third, although somephrases are noisy, such as “not did” and “I ordered,” and some phrasesare misclassified into ratings, such as “new condition” and “new item”being misclassified into the rating 0 class, a majority of the phrasesinformative and indicate the correct ratings. In addition, the frequencycounts allow users to determine whether these opinions arerepresentative of the major opinions. Fourth, there is some correlationbetween the predicted aspect ratings and the phrase frequency counts.Usually, a high aspect rating maps to a large number of phrases and arating of 1 while a low aspect rating maps to a small number of phrasesand a rating of 0.

A sample comparison of two sellers is shown in Table 3. For brevity,only a part of the summary is displayed. Although the two sellers have avery similar overall rate (98.66% positive, versus 98.16% positive),seller 1 is better at providing good shipping while seller 2 is strongerat good communication. This table provides more detailed informationthan the overall ratings and shows one benefit of decomposing an overallrating into aspect ratings.

TABLE 3 Sample Comparison of Two Sellers Aspects Seller1 Seller2 OVERALL98.66% 98.16% described 4.7967 4.8331 communication 4.5956 4.9462shipping 4.9131 4.2244

An aspect evaluator 502 is used to quantitatively evaluate theeffectiveness of aspect discovery and clustering. The aspect evaluator502 measures both aspect coverage and clustering accuracy.

To evaluate the aspects and clusters generated by the aspect module 204of FIG. 3, a standard dataset is created by users. For each seller andno more than 100 head terms are displayed that each, in turn, havesupport for no less than 0.1% of the comments (for typical seller, thereare about 80 terms). The term frequency and five most frequent phrasesare also displayed. The example for the aspect “price” is 608, 0.012,“great price,” “good price,” “fair price,” “nice price,” and “reasonableprice.” More specifically, the head term is “price,” which appears 608times in the seller's feedback comments (i.e., term count), accountingfor 1.2% of all head terms (i.e., term frequency). The most frequentphrases with this head term are “great price,” “good price,” “fairprice,” “nice price,” and “reasonable price.” These phrases aredisplayed, in part, to provide the user with some context for clusteringthe head terms in case there is any ambiguity. The users are then askedto cluster the head terms into no more than eight clusters based ontheir meanings. If more than eight clusters are formed, the userpreserves the top eight clusters with highest support. Some head termsthat do not look like aspects (e.g., because of parsing errors) or thatdo not fit into top eight clusters are ignored.

Aspect coverage is used to measure how much an aspect clusteringalgorithm is able to recover the major aspects that the users haveidentified. If the most frequent term in an algorithm output clustermatches one of the terms in the human identified cluster, it is countedas an aspect match. Top K clusters are the K clusters of the largestsize. Then, aspect coverage at top K is defined as the number of aspectmatches within top K clusters divided by K.

However, aspect coverage may only evaluate the most frequent term ineach cluster (e.g., as the label of a cluster). Aspect coverage does notmeasure the coherence of terms within the cluster. Therefore, aclustering accuracy algorithm is used to measure the clusteringcoherence performance. Given a head term w_(h), let h(w_(h)) andA(w_(h)) be the human annotated cluster label and the label generated bysome algorithm, respectively. The clustering accuracy is defined asfollows:

${{Clustering}\mspace{14mu}{Accuracy}} = \frac{\Sigma_{w_{h} \in V_{h}}\mspace{14mu}{\delta\left( {{h\left( w_{h} \right)},{{map}\left( {A\left( w_{h} \right)} \right)}} \right)}}{\left| V_{h} \right|}$where |V_(h)| is the total number of head terms, δ(x,y) is the deltafunction that equals 1 if x=y and equals zero otherwise, andmap(A(w_(h))) is the permutation mapping function that maps each clusterlabel A(w_(h)) to the equivalent label from the human annotation. Onesuch mapping function is the Kuhn-Munkres algorithm.

The three aspect clustering methods (k-means, unstructured PLSA, andstructured PLSA) are compared using aspect coverage and clusteringaccuracy. FIG. 6 depicts a chart of the aspect coverage in oneevaluation of the dataset. As shown in FIG. 6, both probabilisticmodels, i.e., unstructured PLSA and structured PLSA, are able toidentify a small number of the most significant aspects (when K issmall). As the number of clusters increases, the performances of thethree methods converge to a similar level, around 0.8. This indicatesthat all of the three methods are able to identify the eight majoraspects reasonably well. Table 4, below, includes the results of thecluster accuracy algorithm for each of the aspect clustering methods.

TABLE 4 Evaluation of Cluster Accuracy Method Clustering Accuracyk-means 0.36 Unstructured PLSA 0.32 Structured PLSA 0.52Based on table 4, the structured PLSA method achieves the bestperformance of clustering accuracy, 0.52, meaning that the clusters aremost coherent with respect to user-generated clusters.

Additionally, the degree to which users agree on the coherence inclustering tasks was measured to establish an “upper bound” performance.In this measurement, three users are asked to label the same set ofthree sellers. Then, the user agreement is evaluated as the clusteringaccuracy between each pair of users, as shown in table 5 below.

TABLE 5 Human Agreement on Clustering Accuracy Seller1 Seller2 Seller3AVG Annot1-Annot2 0.6610 0.5484 0.6515 0.6203 Annot1-Annot3 0.78460.6806 0.7143 0.7265 Annot2-Annot3 0.7414 0.6667 0.6154 0.6745 AVG0.7290 0.6319 0.6604 0.6738It can be seen that human agreement can vary from 0.5484 to 0.7846,across different annotator pairs and different data. The averageagreement is 0.6738. The human agreement curve is plotted with differentcutoffs of head term support values in FIG. 7. A higher support valueresults in a smaller number of head terms. In the three curves ofclustering accuracy, denoting three pairs of annotators, the dataconverges to 1 at a point of support value 5.5%, where there are onlythree or four terms remaining. Before the point of minimum support, mostagreement is still no more than 0.8.

The rating evaluator 504 of FIG. 5 evaluates the rating predictionsgenerated by the rating module 206 of FIG. 4 using an aspect rankingcorrelation and ranking loss. The evaluation compares the rankings todetailed seller ratings (DSRs) submit by buyers. The descriptions forthe four DSR criteria are used as priors when estimating the four aspectmodels, so that the discovered aspects align with the DSR criteria.Then, the predicted ratings are mapped into a range of [0, 5] in orderto allow comparison with the actual DSR ratings provided by buyers. Notethat the algorithms do not use any information from the true DSRratings. Instead, the DSR ratings are predicted based on the comments inthe overall ratings. If the algorithms are accurate, the predictions areexpected to be similar to the true DSR ratings by the buyers who wrotethe comments.

Since the aspect rating prediction also depends on the quality of aspectclusters, the two methods of rating prediction (local prediction andglobal prediction) are compared using the three different aspectclustering programs. While there is no easy way to incorporate suchprior information into the k-means clustering algorithm, the k-meansclusters are mapped to four DSR criteria as a post-processing step. Todo so, the k-means clusters are aligned to make DSR if that clustercontains the description word of the DSR, if such alignment is not in aDSR, a cluster is randomly selected. A baseline is also included in thecomparison using the positive feedback percentage to predict each aspectwithout extracting aspects from the comments.

To measure the effectiveness of ranking the four DSRs for a givenseller, the aspect ranking correlation is determined. For example, aseller may be better at “shipping” than at “communication.” BothKendall's Tau rank correlation and Pearson's correlation coefficient maybe used. Ranking loss measures the average distance between the true andpredicted rankings. The ranking loss for an aspect is expressed as:

$\sum\limits_{i}\mspace{14mu}\frac{\left| {{{actual}_{—}{rating}_{i}} - {{predicted}_{—}{rating}_{i}}} \right|}{N}$where N=28 is the number of sellers. Average ranking loss on K aspectsis the average over each aspect. The results are shown in table 6,below, and the best performance of each column is marked in bold font.

TABLE 6 Evaluation Results on Aspect Rating Prediction Aspect AspectCorrelation Ranking Loss Clustering Prediction Kendal's Tau Pearson DSR1DSR2 DSR3 DSR4 AVG of 4 AVG of 3 baseline 0.2892 0.3161 0.1703 0.2053 03332 0.4372 0.2865 0.2363 k-means Local Prediction 0.1106 0.1735 0.14690.1925 0 3116 0.4177 0.2672 0.2170 k-means Global Prediction 0.1225−0.0250 1.3954 0.2726 0.2242 0.3750 0.5668 0.6307 Unstructured PLSALocal Prediction 0.2815 0.4158 0.1402 0.1439 0.3092 0.3514 0.2362 0.1977Unstructured PLSA Global Prediction 0.4958 0.5781 0.2868 0.1262 0.21720.4228 0.2633 0.2101 Structured PLSA Local Prediction 0.1905 0.45170.1229 0.1386 0.3113 0.3420 0.2287 0.1909 Structured PLSA GlobalPrediction 0.4167 0.6118 0.0901 0.1353 0.2349 0.5773 0.2594 0.1534

A good prediction is identified by a high correlation and a low rankingloss. As is apparent, the aspect clustering quality affects theprediction of aspect ratings. For example, if k-means is used to clusterthe aspects, no matter which prediction algorithm is used, theprediction performance is poor, being below the baseline performance andespecially for correlation. Further, the global prediction algorithmperforms better than the local prediction algorithm at correlation forboth unstructured and structured PLSA aspect clustering. This indicatesthat the ratings predicted by global prediction are more discriminativeand accurate in ranking the four DSRs. Moreover, the ranking lossperformance of the methods described for unstructured PLSA andstructured PLSA and local prediction or global prediction is almostalways better than the baseline. The best ranking loss averaged amongthe four DSRs is 0.2287 given by structured PLSA and local predictioncompared to the baseline of 0.2865. Furthermore, the ranking lossperformance also varies across different DSRs. For example, thedifference is most significant on DSR 4, which is “shipping and handlingcharges.” However, the problem is that the term “charges” almost neveroccurs in the comments, so that the aspect clusters estimated using thisprior is kind of randomly related to “shipping and handling charges,”resulting in the low performance on the prediction for this aspect. Ifthis aspect is excluded in the average is taken of the other threeranking losses, average ranking loss performance of each algorithmimproves the best performance is achieved by structured PLSA and globalprediction at 0.1534 compared with 0.2365 by the baseline.

The extraction evaluator 506 of FIG. 5 evaluates the precision andrecall of the phrases extracted by the extraction module 208 of FIG. 2.To generate a standard for representative phrases, both the true DSRratings and human annotations are used. The DSR ratings are used togenerate candidate phrases at different rating levels based on theassumption that if a buyer provided a low rating (e.g. less than orequal to 3 out of 5) on an aspect, the user will express a negativeopinion for this aspect in the text comments. In order to rule out thebias from the aspect clustering algorithm, aspects for the phrases arenot distinguished when displaying the phrases to the users. Tosummarize, the comments with low DSR ratings and high DSR ratings areaggregated separately, and the most frequent 50 phrases in each set aredisplayed. The user is asked to select the three most frequent phrasesfor opinions of rating 1 and rating 0 on each of the four aspects. An

TABLE 7 Sample Representative Phrases by Human Annotation DSR CriteriaPhrases of Rating 1 Phrases of Rating 0 ITEM AS as described (15609)than expected (6) DESCRIBED as promised (1282) as expected 487 COMMUN-great communication (1164) poor communication (22) ICATION goodcommunication (1018) bad communication (12) excellent communication(266) SHIPPING fast shipping (28447) slow shipping (251) TIME fastdelivery (3919) slow delivery (20) quick shipping (3812) not ship (18)SHIPPING excessive postage (10) AND HANDLING CHARGESexample output from the human annotation is shown below in Table 7.The user is given a list of candidates for rating 1 phrases and a listof candidates for rating 0 phrases, and is then asked to fill in theeight cells, as shown in Table 7. In some cases, there are no phrasesthat fit into some cell. In this instance, the cell is simply left hisempty. For example, there are no positive phrases for “shipping andhandling charges.”

The representative phrases, extraction algorithm is applied in additionto different aspect clustering and rating prediction algorithms, andoutputs three phases for each of the eight cells in Table 7. Each cellis treated as a “query,” human generated phrases are treated as“relevant documents,” and computer generated phrases as “retrieveddocuments.” The precision and recall are calculated to evaluate theinformation retrieval:

${Precision} = \frac{\left| {\left\{ {{relevant}_{—}{docs}} \right\}\bigcap\left\{ {{docs}_{—}{retrieved}} \right\}} \right|}{\left| \left\{ {{docs}_{—}{retrieved}} \right\} \right|}$${Recall} = \frac{\left| {\left\{ {{relevant}_{—}{docs}} \right\}\bigcap\left\{ {{docs}_{—}{retrieved}} \right\}} \right|}{\left| \left\{ {{relevant}_{—}{docs}} \right\} \right|}$The average precision and average recall are reported in Table 8 basedon the user annotation of 10 sellers.

TABLE 8 Evaluation of Representative Phrases Methods Prec. Recallk-means + Local Prediction 0.3055 0.3510 k-means + Global Prediction0.2635 0.2923 Unstructured PLSA + Local Prediction 0.4127 0.4605Unstructured PLSA + Global Prediction 0.4008 0.4435 Structured PLSA +Local Prediction 0.5925 0.6379 Structured PLSA + Global Prediction0.5611 0.5952Note that when the user fills out the cells in the table, the user isalso classifying the phrases into the four aspects and removing thephrases that are not associated with the correct rating. As such, phraseextraction is also an indirect way of evaluating the aspect clusteringand aspect rating prediction algorithms. Based on Table 8, no matterwhich of the rating prediction algorithms is used, structured PLSAconsistently outperforms unstructured PLSA, and both of which arepreferable to k-means. Second, local prediction consistently outperformsglobal prediction independent of the underlying aspect clusteringalgorithm. This indicates that local prediction is sufficient and mayeven be better than global prediction at selecting only a fewrepresentative phrases for each aspect. The best performance is achievedby structured PLSA and local prediction at an average precision of0.5925 and average recall of 0.6379.

FIG. 8 is a flowchart of a technique 800 for summarizing the shortcomments according to various embodiments. In a step 802, the aspects ofshort comments are identified and clustered by, for example, the aspectmodule 204 of FIG. 3 using k-means, unstructured PLSA, and/or structuredPLSA. In a step 804, ratings are predicted for each aspect (or aspectcluster) from the overall rating by, for example, the rating module 206of FIG. 4 using local prediction and/or global prediction. In a step806, representative phrases are extracted from the short comments by,for example, the extraction module 208 of FIG. 2. Optional step 808includes evaluating the results by, for example, the evaluation module210 of FIG. 5.

FIG. 9 is a network diagram depicting a client-server system 900, withinwhich one example embodiment may be deployed. A networked system 902, inthe example forms of a network-based marketplace or publication system,provides server-side functionality, via a network 904 (e.g., theInternet or Wide Area Network (WAN)) to one or more clients. FIG. 9illustrates, for example, a web client 906 (e.g., a browser, such as theInternet Explorer browser developed by Microsoft Corporation of Redmond,Wash. State), and a programmatic client 908 executing on respectiveclient machines 910 and 912.

An Application Program Interface (API) server 914 and a web server 916are coupled to, and provide programmatic and web interfaces respectivelyto, one or more application servers 918. The application servers 918host one or more marketplace engines 920 and summarization engines 922.The application servers 918 are, in turn, shown to be coupled to one ormore databases servers 924 that facilitate access to one or moredatabases 926.

The marketplace engines 920 may provide a number of marketplacefunctions and services to users that access the networked system 902.The summarization engines 922 may likewise provide summarizationservices to users. While the marketplace and summarization engines 920and 922 are shown in FIG. 9 to both form part of the networked system902, it will be appreciated that, in alternative embodiments, thesummarization engines 922 may form part of a summarization service thatis separate and distinct from the networked system 902.

Further, while the system 900 shown in FIG. 9 employs a client-serverarchitecture, the present invention is, of course, not limited to suchan architecture, and could equally well find application in adistributed, or peer-to-peer, architecture system, for example. Thevarious marketplace and payment applications 920 and 922 could also beimplemented as standalone software programs, which do not necessarilyhave networking capabilities.

The web client 906 accesses the various marketplace and summarizationengines 920 and 922 via the web interface supported by the web server916. Similarly, the programmatic client 908 accesses the variousservices and functions provided by the marketplace and summarizationengines 920 and 922 via the programmatic interface provided by the APIserver 914. The programmatic client 908 may, for example, be a sellerapplication (e.g., the TurboLister application developed by eBay Inc.,of San Jose, Calif.) to enable sellers to author and manage listings onthe networked system 902 in an off-line manner, and to performbatch-mode communications between the programmatic client 908 and thenetworked system 902.

FIG. 9 also illustrates a third party application 928, executing on athird party server machine 930, as having programmatic access to thenetworked system 902 via the programmatic interface provided by the APIserver 914. For example, the third party application 928 may, utilizinginformation retrieved from the networked system 902, support one or morefeatures or functions on a website hosted by the third party. The thirdparty website may, for example, provide one or more promotional,marketplace or payment functions that are supported by the relevantapplications of the networked system 902.

FIG. 10 shows a diagrammatic representation of machine in the exampleform of a computer system 1000 within which a set of instructions, forcausing the machine to perform any one or more of the methodologiesdiscussed herein, may be executed. In alternative embodiments, themachine operates as a standalone device or may be connected (e.g.,networked) to other machines. In a networked deployment, the machine mayoperate in the capacity of a server or a client machine in server-clientnetwork environment, or as a peer machine in a peer-to-peer (ordistributed) network environment. The machine may be a server computer,a client computer, a personal computer (PC), a tablet PC, a set-top box(STB), a Personal Digital Assistant (PDA), a cellular telephone, a webappliance, a network router, switch or bridge, or any machine capable ofexecuting a set of instructions (sequential or otherwise) that specifyactions to be taken by that machine. Further, while only a singlemachine is illustrated, the term “machine” shall also be taken toinclude any collection of machines that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methodologies discussed herein.

The example computer system 1000 includes a processor 1002 (e.g., acentral processing unit (CPU) a graphics processing unit (GPU) or both),a main memory 1004 and a static memory 1006, which communicate with eachother via a bus 1008. The computer system 1000 may further include avideo display unit 1010 (e.g., a liquid crystal display (LCD) or acathode ray tube (CRT)). The computer system 1000 also includes analphanumeric input device 1012 (e.g., a keyboard), a cursor controldevice 1014 (e.g., a mouse), a disk drive unit 1016, a signal generationdevice 1018 (e.g., a speaker) and a network interface device 1020.

The disk drive unit 1016 includes a machine-readable medium 1022 onwhich is stored one or more sets of instructions (e.g., software 1024)embodying any one or more of the methodologies or functions describedherein. The software 1024 may also reside, completely or at leastpartially, within the main memory 1004 and/or within the processor 1002during execution thereof by the computer system 1000, the main memory1004 and the processor 1002 also constituting machine-readable media.

The software 1024 may further be transmitted or received over a network1026 via the network interface device 1020.

While the machine-readable medium 1022 is shown in an example embodimentto be a single medium, the term “machine-readable medium” should betaken to include a single medium or multiple media (e.g., a centralizedor distributed database, and/or associated caches and servers) thatstore the one or more sets of instructions. The term “machine-readablemedium” shall also be taken to include any medium that is capable ofstoring, encoding or carrying a set of instructions for execution by themachine and that cause the machine to perform any one or more of themethodologies of the present invention. The term “machine-readablemedium” shall accordingly be taken to include, but not be limited to,solid-state memories, and optical and magnetic media.

Thus, a method and system to summarize short comments have beendescribed. The method and system described herein may operate to provideone or more technical solutions to technical problems including, but notlimited to, improved database management, faster access to queryresults, more accurate query results, and providing a better userexperience in the online publication system. Although the presentinvention has been described with reference to specific exampleembodiments, it will be evident that various modifications and changesmay be made to these embodiments without departing from the broaderspirit and scope of the invention. Accordingly, the specification anddrawings are to be regarded in an illustrative rather than a restrictivesense.

The Abstract of the Disclosure is provided to comply with 37 C.F.R. §1.72(b), requiring an abstract that will allow the reader to quicklyascertain the nature of the technical disclosure. It is submitted withthe understanding that it will not be used to interpret or limit thescope or meaning of the claims. In addition, in the foregoing DetailedDescription, it can be seen that various features are grouped togetherin a single embodiment for the purpose of streamlining the disclosure.This method of disclosure is not to be interpreted as reflecting anintention that the claimed embodiments require more features than areexpressly recited in each claim. Rather, as the following claimsreflect, inventive subject matter lies in less than all features of asingle disclosed embodiment. Thus the following claims are herebyincorporated into the Detailed Description, with each claim standing onits own as a separate embodiment.

What is claimed is:
 1. A system comprising: a non-transitory memory tostore instructions and a plurality of comments received from a commentswebsite, each of the plurality of comments related to a target entityand comprising an overall rating related to the target entity and atleast one phrase that includes a head term and a modifier associatedwith the head term that is represented by a bag of phrases; and one ormore processors configured to execute the instructions stored on thememory to cause the system to perform operations comprising: identifyingthe head term in the at least one phrase of a portion of comments of theplurality of comments; mapping the portion of comments to an aspectcorresponding to an attribute of the target entity based on theidentified head term; determining an aspect rating corresponding to theaspect based on an aggregation of the overall ratings in the portion ofcomments, the aspect rating including a numerical measure showing adegree of satisfaction demonstrated in the portion of comments towardsthe aspect; extracting, from the portion of comments, a representativephrase corresponding to each of the aspects that supports or explainsthe aspect rating; generating an output that includes the overallratings of the portion of comments decomposed into two or more of theaspects of the target entity, the aspects having associated therewiththe representative phrases with support information that serve asindices to navigate into a set of specific comments about a particularaspect of the two or more of the aspects; and causing presentation ofthe generated output on a client machine, to tailor comments related tothe target entity to the generated output instead of overall ratings. 2.The system of claim 1, wherein the operations further comprise mappingthe portion of the plurality of comments to the aspect by decomposingthe plurality of comments into a plurality of aspect clusters, whereinthe aspect is included in the plurality of aspect clusters.
 3. Thesystem of claim 1, wherein the head term is identified using a k-meansclustering algorithm.
 4. The system of claim 1, wherein the head term isidentified using an unstructured probabilistic latent semantic analysis(PLSA) algorithm.
 5. The system of claim 1, wherein the head term isidentified using a structured probabilistic latent semantic analysis(PLSA) algorithm.
 6. The system of claim 1, wherein the operationsfurther comprise incorporating a topic model corresponding to theaspect.
 7. The system of claim 1, wherein the operations furthercomprise determining the aspect rating based on respective modifiers ofthe head terms comprising the at least one phrase of respective commentsof the plurality of comments.
 8. The system of claim 1, wherein theoperations further comprise evaluating the at least one phrase using aprecision metric and a recall metric.
 9. A method comprising: receivingusing one or more computer processors, from a comments website, an inputthat includes a plurality of comments related to a target entity, eachcomment of the plurality of comments including an overall rating relatedto the target entity and at least one phrase that includes a head termand a modifier associated with the head term that is represented by abag of phrases; identifying a plurality of aspect clusters based on aplurality of comments, wherein each aspect cluster of the plurality ofaspect clusters corresponds to an aspect of the target entity theidentifying of the plurality of aspect clusters includes: identifyingthe head term in the at least one phrase of a portion of the comments inthe plurality of comments, and mapping each head term to one of aplurality of aspect clusters; determining an aspect rating correspondingto each of the aspect clusters and the corresponding aspect of thetarget entity based on an aggregation of the overall ratings in theportion of the plurality of comments, the aspect rating including anumerical measure showing a degree of satisfaction demonstrated in theportion of comments towards the aspect; extracting, from the portion ofcomments, a representative phrase corresponding of the aspects thatsupports or explains the aspect rating; generating an output thatincludes the overall ratings of the portion of comments decomposed intotwo or more of the aspects of the target entity, the aspects havingassociated therewith the representative phrases with support informationthat serve as indices to navigate into a set of specific comments abouta particular aspect of the two or more of the aspects; and causingpresentation of the generated output on a client machine, to tailorcomments related to the target entity to the generated output instead ofoverall ratings.
 10. The method of claim 9, further comprisingevaluating an aspect coverage of the plurality of aspect clusters. 11.The method of claim 9, further comprising evaluating the plurality ofaspect clusters using a user agreement on clustering accuracy.
 12. Themethod of claim 9, further comprising evaluating the aspect rating usingan aspect rating correlation.
 13. The method of claim 9, furthercomprising evaluating the aspect rating using a ranking loss.
 14. Themethod of claim 9, further comprising evaluating the at least one phraseusing a precision metric and a recall metric.
 15. The method of claim 9,wherein the identifying of the plurality of the aspect clusters isperformed using at least one of k-means clustering, unstructuredprobabilistic latent semantic analysis (PLSA), or structured PLSA. 16.The method of claim 9, wherein the identifying of the plurality of theaspect clusters is based on a topic model corresponding to a respectiveaspect cluster.
 17. A computer-readable storage medium having notransitory signals and embodying instructions executable by one or moreprocessors of a machine, that in response to execution cause the machineto perform operations comprising: receiving from a comments website, aninput that includes a plurality of comments related to a target entity,each comment of the plurality of comments including an overall ratingrelated to the target entity and at least one phrase that includes ahead term and a modifier associated with the head term that isrepresented by a bag of phrases; identifying the head term in the atleast one phrase of a portion of comments of the plurality of comments;mapping each head term to one of a plurality of aspect clusters, whereineach aspect cluster of the plurality of aspect clusters corresponds toan aspect of the target entity and is identified based on the mappedhead terms; determining an aspect rating corresponding to each of theaspect clusters and the corresponding aspect of the target entity basedon an aggregation of the overall ratings in the portion of comments, theaspect rating including a numerical measure showing a degree ofsatisfaction demonstrated in the portion of comments towards the aspect;extracting, from the portion of comments, a representative phrasecorresponding to each of the aspects that supports or explains theaspect rating; generating an output that includes the overall ratings ofthe portion of comments decomposed into two or more of the aspects ofthe target entity, the aspects having associated therewith therepresentative phrases with support information that serve as indices tonavigate into a set of specific comments about a particular aspect ofthe two or more of the aspects; and causing presentation of thegenerated output on a client machine, to tailor comments related to thetarget entity to the generated output instead of overall ratings.